[FLINK-39984][runtime][webUI] Support LITE/FULL thread dump modes by xingsuo-zbz · Pull Request #28732 · apache/flink

xingsuo-zbz · 2026-07-13T09:43:35Z

What is the purpose of the change

Fixes FLINK-39984. Clicking "Thread Dump" on a JobManager or TaskManager in the Web UI can cause the targeted process to miss heartbeats and be marked dead by the JobManager, taking down otherwise healthy running jobs.

The dominant cost is not the RPC dispatch but the underlying JVM safepoint: JvmUtils#createThreadDump unconditionally calls ThreadMXBean.dumpAllThreads(true, true), which requires walking every thread's locked-monitor and j.u.c. synchronizer state inside a single JVM-wide safepoint. On JVMs with many threads (Netty + RocksDB + async I/O + user threads — 10k+ in production is not uncommon) this pauses the entire JVM (including the heartbeat dispatcher) for seconds to tens of seconds. If the pause exceeds heartbeat.timeout (default 50 s), the JM triggers an unnecessary TaskManager failover on what was intended as a purely diagnostic action.

This PR makes the dump granularity configurable so operators can pick a mode whose safepoint pause is bounded well
below heartbeat.timeout. A companion PR under the same JIRA additionally offloads the dump construction to
ioExecutor so the RPC mailbox no longer stalls behind the dump; the two fixes are complementary but independent.

Brief change log

Introduce ThreadDumpMode {LITE, FULL} and thread it through RestfulGateway / TaskExecutorGateway /
ResourceManagerGateway / TaskExecutorGatewayDecoratorBase / NonLeaderRetrievalRestfulGateway.
- LITE → dumpAllThreads(false, false) — stack traces only, negligible JVM pause.
- FULL → dumpAllThreads(true, true) — current behavior, retains locked-monitor / synchronizer info for deadlock analysis.
Expose the mode as an optional REST query parameter ?mode=lite|full on both /jobmanager/thread-dump and /taskmanagers/{id}/thread-dump via a new ThreadDumpModeQueryParameter and two new MessageParameters classes; unknown values become HTTP 400.
Add config option cluster.thread-dump.default-mode (default FULL) that controls the fallback when no query parameter is supplied. Default is deliberately FULL to preserve the existing on-upgrade behavior; the description flags LITE as the recommended value for large clusters. A separate dev@ discussion will decide whether to flip the default in a future release.
Web UI: add a Lite / Full toggle to both thread-dump pages
(flink-runtime-web/web-dashboard/.../{job,task}-manager/thread-dump/). Selecting a mode does not auto-fetch; the user must press the refresh button (avoids surprising the operator when they merely click the toggle). The download link tracks the current selection so exported dumps match the on-screen content.

Verifying this change

This change added tests and can be verified as follows:
New unit tests

ThreadDumpModeQueryParameterTest — 6 cases covering key name, optionality, case-insensitive parsing of LITE/FULL,
unknown value → IllegalArgumentException, and lower-case serialization.
TaskExecutorThreadDumpTest — spins up a real TaskExecutor and verifies that (a) requestThreadDump returns a
non-empty dump for null, LITE, and FULL, and (b) when mode is omitted the request honors
cluster.thread-dump.default-mode (the test overrides it to LITE and asserts the "Number of locked synchronizers"
section is absent, which only appears in FULL).

Existing tests updated

TestingResourceManagerGateway, TestingTaskExecutorGateway, TestingRestfulGateway — signature updates for the new
ThreadDumpMode parameter.
rest_api_v1.snapshot — new mode query parameter recorded for both endpoints; RuntimeRestAPIStabilityTest passes
against the updated snapshot.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (Before this change, requesting a thread dump on a busy TM could cause a spurious Heartbeat of TaskManager timed out. After this change the
request no longer blocks the RPC mailbox and when the operator selects LITE, the
safepoint pause is short enough not to trip heartbeat.timeout.)
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes — the new ThreadDumpMode REST parameter and cluster.thread-dump.default-mode config option.
If yes, how is the feature documented? docs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Claude Code (Claude Opus 4.7)

flinkbot · 2026-07-13T09:48:39Z

CI report:

00cef73 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

spuru9 · 2026-07-13T21:05:51Z

A few minor comments before the review:

The commit message can a co-author reference contradiction the PR guidelines.
Can you fix the test issue. They appear to be genuine issues.

xingsuo-zbz · 2026-07-15T12:28:57Z

@flinkbot run azure

xingsuo-zbz · 2026-07-16T02:29:08Z

@spuru9 done,please continue to review

Myasuka

I think we should split this PR into two PRs, one is to use ioExecutor instead of main thread; one is to introduce new feature that we can have options to select when creating thread dumps.

xingsuo-zbz · 2026-07-21T11:50:15Z

I think we should split this PR into two PRs, one is to use ioExecutor instead of main thread; one is to introduce new feature that we can have options to select when creating thread dumps.

@Myasuka The ioExecutor offload lives in a companion PR#28778 under the same JIRA.

xingsuo-zbz · 2026-07-24T10:35:01Z

@flinkbot run azure

Myasuka · 2026-08-01T16:16:52Z

@xingsuo-zbz Please rebase code after #28778 merged.

ThreadMXBean.dumpAllThreads(true, true) enters a single JVM-wide safepoint to collect monitor/synchronizer state; on busy JVMs the pause can exceed heartbeat.timeout and cause unnecessary TaskManager failover. - Introduce ThreadDumpMode {LITE, FULL}: LITE = dumpAllThreads(false, false), FULL preserves today's (true, true) behavior. Exposed via an optional query parameter `?mode=lite|full` on the JM/TM thread-dump endpoints. - Add config cluster.thread-dump.default-mode (default FULL to preserve upgrade behavior; LITE recommended for large clusters). - Add a Lite/Full toggle to both Web UI thread-dump pages; selecting a mode does not auto-fetch, the download link tracks the selection.

Myasuka

Please take a look at my comments.

Myasuka · 2026-08-03T08:58:37Z

                            "The maximum stacktrace depth of TaskManager and JobManager's thread dump web-frontend displayed.");

+    @Documentation.Section(Documentation.Sections.EXPERT_CLUSTER)
+    public static final ConfigOption<String> THREAD_DUMP_DEFAULT_MODE =


THREAD_DUMP_DEFAULT_MODE is declared as .stringType() with a default of "FULL", and the actual parsing happens later in ThreadDumpMode.fromStringOrDefault, which silently falls back to FULL for any null/blank/unrecognized value.

That fallback behavior is risky specifically for this option: if an operator mistypes the value (e.g. "Ltie"), the config loads without error and the operator has no indication their setting was ignored — the cluster silently keeps using FULL, the exact high-risk mode this PR is meant to let people opt out of.

Could we either:

Switch this to ConfigOptions.key(...).enumType(ThreadDumpMode.class) so invalid values fail fast at config-load time instead of being silently swallowed, or

If we want to keep it as a free-form string (e.g. for forward compatibility), at least log a WARN in fromStringOrDefault/resolve when the configured value doesn't match a known ThreadDumpMode, so this misconfiguration is visible in the logs.

Option 1 seems preferable given ThreadDumpMode is a fixed, closed set of values.

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch 3 times, most recently from bc6b4a8 to 36f9795 Compare July 15, 2026 12:26

Myasuka requested changes Jul 20, 2026

View reviewed changes

Comment thread flink-runtime/src/main/java/org/apache/flink/runtime/rest/messages/ThreadDumpMode.java Outdated

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from 36f9795 to 117a668 Compare July 20, 2026 11:05

xingsuo-zbz changed the title ~~[FLINK-39984][runtime][webUI] Dispatch thread dump to ioExecutor and support SAFE/FULL modes~~ [FLINK-39984][runtime][webUI] Support LITE/FULL thread dump modes Jul 20, 2026

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from 117a668 to 86388ac Compare July 20, 2026 13:09

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from 86388ac to a12f903 Compare July 23, 2026 11:33

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from a12f903 to ebafa89 Compare July 24, 2026 10:38

xingsuo-zbz requested a review from Myasuka July 27, 2026 03:37

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from ebafa89 to be8a20c Compare August 3, 2026 07:00

xingsuo-zbz force-pushed the zbz/thread-dump-safe-mode branch from be8a20c to 00cef73 Compare August 3, 2026 07:25

Myasuka reviewed Aug 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

[FLINK-39984][runtime][webUI] Support LITE/FULL thread dump modes - #28732

[FLINK-39984][runtime][webUI] Support LITE/FULL thread dump modes#28732
xingsuo-zbz wants to merge 1 commit into
apache:masterfrom
xingsuo-zbz:zbz/thread-dump-safe-mode

xingsuo-zbz commented Jul 13, 2026 •

edited

Loading

Uh oh!

flinkbot commented Jul 13, 2026 •

edited

Loading

Uh oh!

spuru9 commented Jul 13, 2026

Uh oh!

xingsuo-zbz commented Jul 15, 2026

Uh oh!

xingsuo-zbz commented Jul 16, 2026

Uh oh!

Myasuka left a comment

Uh oh!

Uh oh!

xingsuo-zbz commented Jul 21, 2026 •

edited

Loading

Uh oh!

xingsuo-zbz commented Jul 24, 2026

Uh oh!

Myasuka commented Aug 1, 2026

Uh oh!

Myasuka left a comment

Uh oh!

Myasuka Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

xingsuo-zbz commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

spuru9 commented Jul 13, 2026

Uh oh!

xingsuo-zbz commented Jul 15, 2026

Uh oh!

xingsuo-zbz commented Jul 16, 2026

Uh oh!

Myasuka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xingsuo-zbz commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xingsuo-zbz commented Jul 24, 2026

Uh oh!

Myasuka commented Aug 1, 2026

Uh oh!

Myasuka left a comment

Choose a reason for hiding this comment

Uh oh!

Myasuka Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xingsuo-zbz commented Jul 13, 2026 •

edited

Loading

flinkbot commented Jul 13, 2026 •

edited

Loading

xingsuo-zbz commented Jul 21, 2026 •

edited

Loading